Validate indexes for CONTROL data #188
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I wonder if we should try to summarize similar problems to avoid returning N times the same output with just a different source name, ex: Index referring to data (11) outside dataset (0) for 82 sources, example:
dataset: CONTROL/SPB_IRU_AGIPD1M1/CTRL/MC1/module2/dBoardTemperature2/timestamp
file: /gpfs/exfel/exp/SPB/201901/p002316/raw/r0174/RAW-R0174-DA02-S00006.h5and add a verbose argument. But that's a different issue. LGTM |
tmichela
approved these changes
May 27, 2021
|
Thanks - I've opened #190 for summarising the issues. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, we were only checking the indexes for INSTRUMENT data, not CONTROL. I think we may have assumed that CONTROL was unlikely to go wrong. On #187, we found a run where there is a problem - CONTROL data missing, INDEX still records it as though it was there - which
extra-data-validatedid not pick up.This extends the same checks we do on INSTRUMENT data to CONTROL, with one difference. We only record the "Index referring to data outside dataset" problem once per source & per file, rather than for every key. When I recorded it for every key, I got a list of some 27000 problems with the run. Once per source should be plenty to point out the problem, and then you can use
h5glanceto find more detail if necessary.